Electronic camera and range sensor systems that provide a measure of distance from the system to a target object are known in the art. Many such systems approximate the range to the target object based upon luminosity or brightness information obtained from the target object. However such systems may erroneously yield the same measurement information for a distant target object that happens to have a shiny surface and is thus highly reflective, as for a target object that is closer to the system but has a dull surface that is less reflective.
A more accurate distance measuring system is a so-called time-of-flight (TOF) system. FIG. 1 depicts an exemplary TOF system, as described in U.S. Pat. No. 6,323,942 entitled CMOS-Compatible Three-Dimensional Image Sensor IC (2001), which patent is incorporated herein by reference as further background material. TOF system 100 can be implemented on a single IC 110, without moving parts and with relatively few off-chip components. System 100 includes a two-dimensional array 130 of pixel detectors 140, each of which has dedicated circuitry 150 for processing detection charge output by the associated detector. In a typical application, array 130 might include 100×100 pixels 230, and thus include 100×100 processing circuits 150. IC 110 also includes a microprocessor or microcontroller unit 160, memory 170 (which preferably includes random access memory or RAM and read-only memory or ROM), a high speed distributable clock 180, and various computing and input/output (I/O) circuitry 190. I/O circuitry can provide functions including analog-to-digital conversion of detection signals, video gain, etc. Among other functions, controller unit 160 may perform distance to object and object velocity calculations.
Under control of microprocessor 160, a source of optical energy 120 is periodically energized and emits optical energy via lens 125 toward an object target 20. Typically the optical energy is light, for example emitted by a laser diode or LED device 120. Some of the emitted optical energy will be reflected off the surface of target object 20, and will pass through an aperture field stop and lens, collectively 135, and will fall upon two-dimensional array 130 of pixel detectors 140 where an image is formed. In some implementations, each imaging pixel detector 140 captures time-of-flight (TOF) required for optical energy transmitted by emitter 120 to reach target object 20 and be reflected back for detection by two-dimensional sensor array 130. Using this TOF information, distances Z can be determined.
Emitted optical energy traversing to more distant surface regions of target object 20 before being reflected back toward system 100 will define a longer time-of-flight than radiation falling upon and being reflected from a nearer surface portion of the target object (or a closer target object). For example the time-of-flight for optical energy to traverse the roundtrip path noted at t1 is given by t1=2·Z1/C, where C is velocity of light. A TOF sensor system can acquire three-dimensional images of a target object in real time. Such systems advantageously can simultaneously acquire both luminosity data (e.g., signal amplitude) and true TOF distance measurements of a target object or scene.
As described in U.S. Pat. No. 6,323,942, in one embodiment of system 100 each pixel detector 140 has an associated high speed counter that accumulates clock pulses in a number directly proportional to TOF for a system-emitted pulse to reflect from an object point and be detected by a pixel detector focused upon that point. The TOF data provides a direct digital measure of distance from the particular pixel to a point on the object reflecting the emitted pulse of optical energy. In a second embodiment, in lieu of high speed clock circuits, each pixel detector 140 is provided with a charge accumulator and an electronic shutter. The shutters are opened when a pulse of optical energy is emitted, and closed thereafter such that each pixel detector accumulates charge as a function of return photon energy falling upon the associated pixel detector. The amount of accumulated charge provides a direct measure of round-trip TOF. In either embodiment, TOF data permits reconstruction of the three-dimensional topography of the light-reflecting surface of the object being imaged. While systems described in U.S. Pat. No. 6,323,942 can acquired Z information, the accuracy of such information is not known on a per frame of acquired data basis.
Some systems determine TOF by examining relative phase shift between the transmitted light signals and signals reflected from the target object. Detection of the reflected light signals over multiple locations in a pixel array results in measurement signals that are referred to as depth images. U.S. Pat. Nos. 6,515,740 (2003) and 6,580,496 (2003) disclose respectively Methods and Systems for CMOS-Compatible Three-Dimensional Imaging Sensing Using Quantum Efficiency Modulation. FIG. 2A depicts an exemplary phase-shift detection system 100′ according to U.S. Pat. No. 6,515,740 and U.S. Pat. No. 6,580,496. Unless otherwise stated, reference numerals in FIG. 2A may be understood to refer to elements identical to what has been described with respect to the TOF system of FIG. 1.
In FIG. 2A, an exciter 115 drives emitter 120 with a preferably low power (e.g., perhaps 50 mW peak) periodic waveform, producing optical energy emissions of known frequency (perhaps a few hundred MHz) for a time period known as the shutter time (perhaps 10 ms). Energy from emitter 120 and detected signals within pixel detectors 140 are synchronous to each other such that phase difference and thus distance Z can be measured for each pixel detector.
The optical energy detected by the two-dimensional imaging sensor array 130 will include amplitude or intensity information, denoted as “A”, as well as phase shift information, denoted as φ. As depicted in exemplary waveforms in FIGS. 2B, 2C, 2D, the phase shift information varies with distance Z and can be processed to yield Z data. For each pulse or burst of optical energy transmitted by emitter 120, a three-dimensional image of the visible portion of target object 20 is acquired, from which intensity and Z data is obtained (DATA'). As described in U.S. Pat. Nos. 6,515,740 and 6,580,496 obtain depth information Z requires acquiring at least two samples of the target object (or scene) 20 with 90° phase shift between emitted optical energy and the pixel detected signals. While two samples is a minimum figures, preferably four samples, 90° apart in phase, are acquired to permit detection error reduction due to mismatches in pixel detector performance, mismatches in associated electronic implementations, and other errors. On a per pixel detector basis, the measured four sample data are combined to produce actual Z depth information data. Further details as to implementation of various embodiments of phase shift systems may be found in U.S. Pat. Nos. 6,515,740 and 6,580,496. However while the systems and methods described in these two patents can acquire Z information, accuracy of the acquired Z information is not known on a per frame basis.
Many factors, including ambient light, can affect reliability of data acquired by TOF systems. The depth accuracy of Z-data information acquired by pixel detectors 140 in array 130 will vary with the amount of optical energy incident on the pixel detector. In extreme cases there can be too little reflected optical energy, for example from dark target objects 20, perhaps black motor vehicle tires, dark clothing on a pedestrian, etc. On the other hand, an overly reflective target object 20 can reflect too much optical energy. An excess of incoming optical energy can saturate the pixel detector, with the result that the Z-data indicates that the target object 20 moved. However the data alone cannot be used to discern whether the target object 20 actually moved, or whether pixel detector saturation has produced erroneous Z-data indicative of movement, when in fact there was no movement. In these and other extreme cases, the Z-data provided by the TOF system can include incorrect depth information. But while the information is incorrect, the prior art provides no mechanism to so identify the questionable Z-data.
As a result, in an attempt to reduce errors, in some TOF systems the transmitted optical energy may be emitted multiple times using different systems settings to increase reliability of the acquired TOF measurements. For example, the initial phase of the emitted optical energy might be varied to cope with various ambient and reflectivity conditions. The amplitude of the emitted energy might be varied to increase system dynamic range. The exposure duration of the emitted optical energy may be varied to increase dynamic range of the system. Further, frequency of the emitted optical energy may be varied to improve the unambiguous range of the system measurements.
In practice, TOF systems may combine multiple measurements to arrive at a final depth image. But if there is relative motion between system 100 or 100′ and target object 20 while the measurements are being made, the TOF data and final depth image can be degraded by so-called motion blur. For example, while acquiring TOF measurements, system 100 may move, and/or target object 20 may move, or may comprise a scene that include motion. If shutter time is 10 ms (25 frames/second), relative motion occurring faster than about 1/40 ms (for a four-sample acquisition) will produce motion blur. As a result, the motion blur will cause erroneous distance Z data, and will yield a depth image with errors. Unfortunately prior art systems do not provide a mechanism to alert the end user of the Z data that the data being processed at any given time may be relatively low confidence data, e.g., data that perhaps should not be relied upon unduly.
Various other patents have provided methods and systems to improve performance of TOF systems. For example, U.S. Pat. No. 6,522,395 entitled Noise Reduction Techniques Suitable for Three-Dimensional Information Acquirable with CMOS-Compatible Image Sensor ICs (2003) is directed to improving noise reduction in time-of-flight systems by integrating the photodetected brightness signal until integration reaches a predetermined threshold level. The system provides a corrected TOF measurement equal to the round-trip time from system to target object plus an additional time needed for the integrated photodetector signal to cross the threshold value, less a constant times the ratio of photodetector signal amplitude after integration over time equal to pulse width of the emitted optical signal, divided by the pulse width duration. U.S. Pat. No. 6,674,895 entitled Methods for Enhancing Performance and Data Acquired from Three-Dimensional Image Systems (2004), and U.S. Pat. No. 6,512,838 entitled Methods for Enhancing Performance and Data Acquired from Three-Dimensional Image Systems (2003) are directed to a time-of-flight system in which accuracy and resolution are enhanced by various techniques including over-sampling the acquired photodetector data and forming running averages with the data. Acquired data may be rejected certain criteria are not met. However TOF systems according to these patents do not per se identify questionable or low confidence data to a user of the system.
Applications for TOF systems can vary, from systems used to implement virtual input devices such as a keyboard (e.g., see U.S. Pat. No. 6,614,422, U.S. Pat. No. 6,710,770), for which Z measurements are over a fairly small range of distance, to vehicular or security or robotic type systems for which the range of applicable Z measurements may be many meters. By way of example, FIG. 3A depicts a prior art system such as 100′ mounted so as to provide the operator of a motor vehicle 200 with information as to Z distance to objects 20 behind the vehicle. Assume that vehicle 200 is backing up, generally towards object 20. System 100′ can obtain Z data and can provide the processed information to an operator-viewable monitor 220. As shown in FIG. 3B, monitor 220 can provide a display 230 with a representation 240 as to Z distance to any proximate target objects. But assume that the target object 20 in FIG. 3A is surrounded by a highly reflective region 210, perhaps a puddle of water. Thus although the actual distance from the rear of vehicle 200 where system 100′ is located to the target object 20 is distance Z5, perhaps due to the reflectivity of the water, the displayed information 240 erroneously reports a smaller distance, here approximately Z1.
Unfortunately although the information being displayed in FIG. 3B is the result of low confidence data, the operator of vehicle 200 has no indication of this infirmity in the data. If system 100′ is allowed to automatically apply the brakes to vehicle 200 when distance Z to a perceived object is too short, perhaps Z<Z2, vehicle 200 will brake needlessly, since the true distance is Z5. A worse case would be where system 100′ erroneously reports the distance to a target object 20 as being greater than the true distance. In such case, since the vehicle operator has no knowledge that the data is questionable, vehicle 200 might continue to back-up until it collides with the target object.
While FIGS. 3A and 3B depict a rear-viewing application of a TOF system 100′, understandably there are other applications, for example robotic systems in a factor, where errors in Z data are very undesirable, especially where the users of the system do not know that erroneous data may be generated.
To recapitulate, prior art TOF systems can, for a variety of reasons, generate erroneous Z-data depth information. Regardless of the TOF application, the Z-data that is acquired cannot by itself be used to determine whether the data is worthy of full confidence. For example, Z data erroneously indicating target object movement from a saturated pixel detector is indistinguishable from Z-data properly indicating target object movement. TOF systems subject to such unidentified ambiguity in the Z-data are adversely affected. Further, in applications such as tracking a target object within a scene, the accuracy of the depth information of each pixel detector in each frame is to be known to identify which pixels to use in calculations.
Unfortunately prior art statistical methods such as spatial standard deviation, which use depth values in neighboring pixel detectors to estimate the depth accuracy, or prior art temporal standards, which use past depth measurements to estimate the depth accuracy, cannot be used to provide assistance. This is because these prior art methods assume that the mean value of the depth, i.e. position of the object(s), does not change. Unfortunately this unchanging positional requirement cannot be achieved in dynamic scenes.
There is a need for a method and system that take into account both accuracy-type positional errors and uncertainty-type positional errors that can exist in a TOF system. Such method and system should detect and identify erroneous or low confidence Z data generated by a TOF system, and quantify accuracy of the acquired depth values in terms of confidence. Preferably such method and system should require no further data than is already acquired by the TOF system.
The present invention provides such a method and system.